feat(meta): initialization and modeling for sql-backend based meta store #12454

yezizp2012 · 2023-09-20T06:42:19Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Related #12449 , this is the very first step of sql-backend based meta store. In this PR, we have modeled and initialized the existing entities except those related to hummock, and implemented some controller logic based on these models, such as SystemParamsController, CatalogController, etc. For other models, you can refer to their implementation to gradually replace other managers.

I added a README.md doc under model_v2 to introduce the steps of defining relations and changes of new model and generate of related files. Cc @zwang28, you can refer to this to model hummock metadata.

Close #12475.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

yezizp2012

Add some comments for better review and as well as guidance on how to modeling and initialization.

yezizp2012 · 2023-09-21T07:54:57Z

src/meta/src/model_v2/actor.rs

@@ -0,0 +1,51 @@
+// Copyright 2023 RisingWave Labs


Defined all models under model_v2 directory.

yezizp2012 · 2023-09-21T07:57:08Z

src/meta/src/model_v2/mod.rs

+pub mod worker_property;
+
+#[derive(Clone, Debug, PartialEq, FromJsonQueryResult, Eq, Serialize, Deserialize, Default)]
+pub struct I32Array(pub Vec<i32>);


Define I32Array struct which can be used in other models. Since only PostgreSQL supports array type, we will define the field as Json and use I32Array instead in the model.

yezizp2012 · 2023-09-21T07:59:24Z

src/meta/src/model_v2/worker.rs

+use sea_orm::entity::prelude::*;
+#[derive(Clone, Debug, PartialEq, Eq, EnumIter, DeriveActiveEnum)]
+#[sea_orm(rs_type = "String", db_type = "String(None)")]
+pub enum WorkerType {


Define enum for model field and store as string in table.

yezizp2012 · 2023-09-21T08:00:34Z

src/meta/src/model_v2/migration/src/m20230908_072257_init.rs

+
+#[async_trait::async_trait]
+impl MigrationTrait for Migration {
+    async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {


Initial version of sql backend, defined tables, foreign keys, indexes and some initial data.

May I ask if this file was generated by sea-orm, or was hand-written?

The migration file itself is generated by sea-orm, but the detailed migration logic is hand-written.

That sounds okay since we don't have to maintain this migration file anymore. 😄

The migration file itself is generated by sea-orm, but the detailed migration logic is hand-written.

That sounds okay since we don't have to maintain this migration file anymore. 😄

Yes, we only need to add some new migration file to include any data correctness, table schema change etc in it, then the system will upgrade automatically. The file won't be too complicate for each version I believe.

yezizp2012 · 2023-09-21T08:01:11Z

src/meta/src/model_v2/migration/src/m20230908_072257_init.rs

+                            .boolean()
+                            .not_null(),
+                    )
+                    .foreign_key(


For SQLite, foreign key can only be defined along with the table.

yezizp2012 · 2023-09-21T08:02:23Z

src/meta/src/model_v2/migration/src/m20230908_072257_init.rs

+                    )
+                    .col(ColumnDef::new(Fragment::StreamNode).json().not_null())
+                    .col(ColumnDef::new(Fragment::VnodeMapping).json())
+                    .col(ColumnDef::new(Fragment::StateTableIds).json())


For example, defines json type here and derived to array in the model of table fragment.

yezizp2012 · 2023-09-21T08:03:10Z

src/meta/src/model_v2/migration/src/m20230908_072257_init.rs

+}
+
+#[derive(DeriveIden)]
+enum Cluster {


Define each table as enum and use it in migration.

yezizp2012 · 2023-09-21T08:12:52Z

src/meta/src/rpc/server.rs

@@ -352,12 +378,31 @@ pub async fn start_service_as_election_leader(
    mut svc_shutdown_rx: WatchReceiver<()>,
 ) -> MetaResult<()> {
    tracing::info!("Defining leader services");
+    if let Some(sql_store) = &meta_store_sql {


Check and do auto upgrade when meta starts. It will run any migration plans if found in this version. It will ignore if all plans are applied already.

codecov · 2023-09-21T12:52:11Z

Codecov Report

Merging #12454 (bea7128) into main (0126304) will increase coverage by 0.03%.
Report is 1 commits behind head on main.
The diff coverage is 73.94%.

@@            Coverage Diff             @@
##             main   #12454      +/-   ##
==========================================
+ Coverage   69.37%   69.41%   +0.03%     
==========================================
  Files        1440     1469      +29     
  Lines      238947   240650    +1703     
==========================================
+ Hits       165762   167038    +1276     
- Misses      73185    73612     +427

Flag	Coverage Δ
rust	`69.41% <73.94%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
src/meta/src/model_v2/migration/src/lib.rs	`100.00% <100.00%> (ø)`
src/meta/src/model_v2/migration/src/main.rs	`0.00% <0.00%> (ø)`
src/meta/src/model_v2/mod.rs	`0.00% <0.00%> (ø)`
src/meta/src/model_v2/system_parameter.rs	`50.00% <50.00%> (ø)`
src/common/src/system_param/mod.rs	`84.13% <50.00%> (ø)`
src/meta/src/model_v2/cluster.rs	`0.00% <0.00%> (ø)`
src/meta/src/model_v2/election_leader.rs	`0.00% <0.00%> (ø)`
src/meta/src/model_v2/election_member.rs	`0.00% <0.00%> (ø)`
src/meta/src/model_v2/object_dependency.rs	`0.00% <0.00%> (ø)`
src/meta/src/lib.rs	`0.00% <0.00%> (ø)`
... and 25 more

... and 14 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

shanicky

LGTM, we need more test in the future

BugenZhao · 2023-09-25T08:47:02Z

Will review later. 🥵

BugenZhao

🥵🚀🤯

src/common/src/system_param/mod.rs

src/meta/src/controller/catalog.rs

src/meta/src/model_v2/actor.rs

BugenZhao · 2023-09-25T10:14:26Z

src/meta/src/model_v2/migration/src/m20230908_072257_init.rs

+
+#[async_trait::async_trait]
+impl MigrationTrait for Migration {
+    async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {


May I ask if this file was generated by sea-orm, or was hand-written?

src/meta/src/controller/catalog.rs

BugenZhao · 2023-09-25T11:12:55Z

BTW, have we investigated diesel ORM or directly using sqlx? What's the comparison between them?

yezizp2012 · 2023-09-25T11:17:00Z

BTW, have we investigated diesel ORM or directly using sqlx? What's the comparison between them?

diesel is non-async right now. Using sea-orm is to be compatible with different backends and avoid writing duplicate logic for different backends.

BugenZhao · 2023-09-26T06:30:53Z

BTW, have we investigated diesel ORM or directly using sqlx? What's the comparison between them?

diesel is non-async right now. Using sea-orm is to be compatible with different backends and avoid writing duplicate logic for different backends.

I see.

diesel-async is experimental, where sqlite is not supported.
Although sqlx supports any driver, it doesn't seem to handle the dialects between different systems.

src/meta/src/controller/catalog.rs

fuyufjh

Generally LGTM

src/meta/src/controller/catalog.rs

src/meta/src/controller/system_param.rs

BugenZhao

LGTM!!!!!!!!

yezizp2012 added 21 commits September 11, 2023 14:08

feat: init relation and migration

9884181

add model

e1537f5

add init version of migration

d232ce8

init catalog controller

a9bfed0

fix and some todo

382dc5c

Merge branch 'main' into feat/sql-backend-meta

ecdf955

add system parameter and cluster info/ctl, refine catalog ctl

2abc6b2

update models

67b3a3b

add models

d43c58e

add find all downstream objects

98cdaaf

fix drop database

847ac76

rename

20dc757

Merge branch 'main' into feat/sql-backend-meta

52b6feb

apply sql meta store for system params and telemetry

14496b2

Merge branch 'main' into feat/sql-backend-meta

799133c

support sqlite

49e5e88

add election tables

9f0491b

add create schema

d4baff9

update some fields

ec5cc08

add election tables

99d6354

Merge branch 'main' into feat/sql-backend-meta

4aa0fe4

github-actions bot added the type/feature label Sep 20, 2023

This comment was marked as resolved.

Sign in to view

yezizp2012 and others added 2 commits September 20, 2023 06:43

Fix "cargo-hakari"

dcbb94e

ignore license checker for new models

db605b8

yezizp2012 force-pushed the feat/sql-backend-meta branch 2 times, most recently from 66a9b0d to db605b8 Compare September 20, 2023 07:35

yezizp2012 and others added 3 commits September 20, 2023 15:56

Merge branch 'main' into feat/sql-backend-meta

97e93d1

Fix "cargo-hakari"

1166621

use enum and i32 array instead in the models

7e35fb5

yezizp2012 requested review from BugenZhao and zwang28 September 21, 2023 07:40

yezizp2012 added 3 commits September 21, 2023 15:42

fix

cde742e

fix

1965d46

auto upgrade in leader only

c3043ec

yezizp2012 commented Sep 21, 2023

View reviewed changes

yezizp2012 added 2 commits September 21, 2023 16:28

clippy

3dd9fdd

disabled in madsim

235bcd1

shanicky approved these changes Sep 25, 2023

View reviewed changes

BugenZhao reviewed Sep 25, 2023

View reviewed changes

address comments

196867d

Merge branch 'main' into feat/sql-backend-meta

6b9c067

BugenZhao reviewed Sep 26, 2023

View reviewed changes

src/meta/src/controller/catalog.rs Outdated Show resolved Hide resolved

BugenZhao requested a review from fuyufjh September 26, 2023 06:47

add doctest

d1846b5

fuyufjh approved these changes Sep 26, 2023

View reviewed changes

src/meta/src/controller/catalog.rs Show resolved Hide resolved

src/meta/src/controller/system_param.rs Show resolved Hide resolved

BugenZhao approved these changes Sep 26, 2023

View reviewed changes

yezizp2012 added 2 commits September 27, 2023 13:50

Merge branch 'main' into feat/sql-backend-meta

e8bb178

Merge branch 'main' into feat/sql-backend-meta

bea7128

yezizp2012 enabled auto-merge September 27, 2023 08:29

yezizp2012 added this pull request to the merge queue Sep 27, 2023

Merged via the queue into main with commit 6f9c5c2 Sep 27, 2023
7 of 8 checks passed

yezizp2012 deleted the feat/sql-backend-meta branch September 27, 2023 08:55

zwang28 mentioned this pull request Oct 8, 2023

feat(meta): define model V2 for hummock metadata #12674

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(meta): initialization and modeling for sql-backend based meta store #12454

feat(meta): initialization and modeling for sql-backend based meta store #12454

yezizp2012 commented Sep 20, 2023 •

edited

Loading

This comment was marked as resolved.

yezizp2012 left a comment

yezizp2012 Sep 21, 2023

yezizp2012 Sep 21, 2023

yezizp2012 Sep 21, 2023

yezizp2012 Sep 21, 2023

BugenZhao Sep 25, 2023

BugenZhao Sep 26, 2023

yezizp2012 Sep 26, 2023

yezizp2012 Sep 21, 2023

yezizp2012 Sep 21, 2023

yezizp2012 Sep 21, 2023

yezizp2012 Sep 21, 2023

codecov bot commented Sep 21, 2023 •

edited

Loading

shanicky left a comment

BugenZhao commented Sep 25, 2023

BugenZhao left a comment

BugenZhao Sep 25, 2023

BugenZhao commented Sep 25, 2023 •

edited

Loading

yezizp2012 commented Sep 25, 2023

BugenZhao commented Sep 26, 2023

fuyufjh left a comment

BugenZhao left a comment

feat(meta): initialization and modeling for sql-backend based meta store #12454

feat(meta): initialization and modeling for sql-backend based meta store #12454

Conversation

yezizp2012 commented Sep 20, 2023 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

This comment was marked as resolved.

yezizp2012 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 21, 2023 • edited Loading

Codecov Report

shanicky left a comment

Choose a reason for hiding this comment

BugenZhao commented Sep 25, 2023

BugenZhao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BugenZhao commented Sep 25, 2023 • edited Loading

yezizp2012 commented Sep 25, 2023

BugenZhao commented Sep 26, 2023

fuyufjh left a comment

Choose a reason for hiding this comment

BugenZhao left a comment

Choose a reason for hiding this comment

yezizp2012 commented Sep 20, 2023 •

edited

Loading

codecov bot commented Sep 21, 2023 •

edited

Loading

BugenZhao commented Sep 25, 2023 •

edited

Loading